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Ls INTRODUCTION 


A. MOTIVATION 


Software is essential to almost every facet of our 
daily lives from business to science, and while the 


advantages are numerous and have arguably bettered our 





lives, it comes with a cost. The National Institute of 








Standard and Technology (NIST) sponsored a study in 2001 and 
found that the annual cost of software errors to the U.S. 
Economy in 2001 was approximately S59" 5 billovon.+ 
Additionally the study found that over half the costs have 


been borne by the users. This is remarkable because software 








practitioners have not yet been held accountable to the same 








standards imposed on engineers in traditional engineering 


disciplines. 


Over the years, some software defects have resulted in 


human injuries, property damage, and in extreme cases, loss 





of human lives. This is a cost that is unacceptable to users 
and must not be accepted by software developers. One of the 
most well known examples of software error causing human 
fatalities is the THERAC 254. This machine was supposed to 
save human lives by sending the proper amount of radiation 


into patients, but instead it overdosed humans with massive 


1 National Institute of Standards and Technology (NIST), “Software 
errors cost U.S. economy $59.5 billion annually.” National Institute of 
Standards and Technology (NIST2002-10) (2002), 
http://www.nist.gov/public_affairs/releases/n02-10.htm (accessed May 10, 
2008). 


2 N. Levenson and C. Turner, “Investigation of the Therac 25 
accidents.” IEEE Computer (July 1993), 18-41. 
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amounts of radiation and resulted in several fatalities.3 
These failures are not just unacceptable but could have been 


avoided had the software been validated and verified 





properly. The TEEE standards, for validation and 


verification, help to guide the software developers with two 








main questions, “am I building the right product?” and “am 





building the product right?” These questions are 





longstanding and will, if answered appropriately, help 


reduce the risk of mishaps due to software defects. 


Another challenge the software industry faces is that 





software is increasing in complexity, making it difficult to 








detect errors and eliminate them. This also increases the 





importance of validation and testing methods that enable 








earlier and mor ffectiv rror identification and removal. 





Software must be verified and validated to ensure not just 
quality and safety but also guard against waste, in terms of 


money and lives. 


Our motivation for the research reported her is to 











develop techniques that improve th ngineer’s ability to 





validate software systems. Additionally, these validation 
techniques will be applicable to the entire software 


industry and when used in conjunction with verification will 





hopefully result in better software systems and reduce 





software defects and their attendant costs. 








3 R, Merritt, “Embedded experts: fix code bugs or cost lives.” 
Information Week (April 10, 2006), http://www.informationweek.com/news/ 
management /showArticle. jhtml?articleID=185300011 (accessed May 05, 
2008). 


B. INDEPENDENT VALIDATION AND VERIFICATION 


The current guideline for validation and verification 





is the IEEE standard 1012-20044 which has had to evolve 





because of decades of unsuccessful software. The overall aim 
is to establish guidelines for the software industry to 


follow and help to create better software products. 





The following is directly quoted from the IEEE standard 





1012-2004°: 


Software V&V processes consists of the following: 








e Verification process and validation process. The 
verification process provides objectiv videnc 
whether the software and its associated products and 
processes conform to requirements (e.9., for 


correctness, completeness, consistency, accuracy) for 
all life cycle activities during each life cycle 
process acquisition, supply, development, operation, 
and maintenance) satisfy standards, practices, and 
conventions during life cycle processes. 





e Successfully complete each life cycle activity and 
satisfy all the criteria for initiating succeeding life 
cycle activities (e.9g., building the software 
correctly). 

















e The validation process provides evidence whether the 
software and its associated products and processes 
satisfy system requirements allocated to software at 
the end of each life cycle activity. 





e Solve the righ 
laws, implement 
assumptions). 


problem (e.g., correctly model physical 
business rules, use the proper system 








e Satisfy intended use and user needs. 





4 Institute of Electrical and Electronics Engineers, Standard for 
Software Verification and Validation, IEEE-STD-1012, June 08, 2005. 





























5 Institute of Electrical and Electronics Engineers, Standard for 
Software Verification and Validation, IEEE-STD-1012, June 08, 2005. 


) 
































The verification process and the validation process are 


interrelated and complementary processes that use each 





other’s process results to establish better completion 
criteria and analysis, evaluation, review, inspection, 
assessment, and test VéV tasks for each software life cycle 


activity. 





The IEEE guidelines leav th software industry to 








develop their own validation and verification methods. The 


industry has yet to integrat thes techniques fully, 











contributing to the poor record of software acquisition. In 





fact, the Standish Group reported that the success rate of 





software projects was 35% in 2006, and the report claimed 








that software developers fielded only 46% of the required 





features and functions, which means that the projects did 


not meet the needs of the customer.® These studies indicate 








conformance to the V&V guidelines are not enough to 


Significantly lower the defect rate in software partition of 











systems. The V&V guidelines empower the individual to create 








and implement their own IV&V plan of actions, and the only 





consequence to not following the guidelines is unsuccessful 
acquisition of software. Formal V&V (FV&V) techniques can be 
used to address some of the failings of the existing 


practice of V&V. 


Cc. THE NEED FOR A STANDARD TECHNIQUE 


The current problem facing the software industry in 


facilitating validation is that there are no concise, simple 











techniques to conduct validation. The IEEE IV&V guidelines 





are general and meant to guide the industry in the actual 





6 The Standish Group International, "Annual Chaos Report." (2006). 
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implementation of validation. This leaves the software 








industry to its own devices on the implementation of 


validation and can result, in the worst case scenario, with 


if validation is conducted at all. 








poor validation processes 








This is largely due to the ignorance of validation 
procedures and misunderstanding of the necessity of 
validation. And it has resulted in the industry’s primary 


focus on verification because it can be easier to accomplish 


and minimal effort is put into validation. Thus much effort 


is in “have we built the system right?” but not “have we 


built the right system?” 


Adding to the problem, there is no general consensus 


among the academic community on how to complete the 





validation phase, many different paths are used. The typical 


if there 





way for a system to be built, is structure at all, 





is for the software requirements to be gathered, using pen 





and paper, use cases built and then code is written directly 


from the use cases. There is no formal link from the 


requirements to the formal specification of the system 





behaviors (if one exists) to ensure that the correct system 


is being built. The formal specification of system behaviors 





includes assertions which precisely model the required 


behavior of the system and can be traceable 


to the system 


requirements providing a means to ensure 


system is being built. As examples, 


Galileo had significant software errors; 





of 





had not been 








can Significantly reduce thes 





identified or developed by the developers. 











that the correct 


both Voyager and 


the primary cause 


the faults were directly related to system behaviors that 


One 


kids of errors by formally 








specifying the required behaviors in terms of assertions and 
validating the correctness of these assertions against 


stakeholder expectations before building the software. 





One way to facilitate the validation process is through 





execution-based validation. Execution-based validation is 





the process of inferring certain behavioral properties to 





xercise the system under test (SUT) in a known environment 





and with selected inputs. This gives the person conducting 


validation the capability to validate that the system being 





built is the correct system based on user requirements. In 





our thesis we use the StateRover white-box automatic test- 


generator. The white-box test generator constructs a JUnit 








TestCase class from a given statechart assertion model and 
the associate embedded assertions. The advantages of this 
process include: the ability to pinpoint specific errors; 
investigate the causes of failures on a specific input in 


detail; and eliminate errors in their design in an efficient 





manner. 


D. THE ROLE OF SOFTWARE REUSE 


Software reuse is an important concept that can help 
clarify validation techniques, making them more relevant for 


software development teams. Software reuse aims to increase 





the productivity, efficiency and quality of software by 
reusing the applicable software from one project in another 


project.’ By reusing the software the developers can save 








resources that would have otherwise been used to develop the 


software. 








7 We. Lim, Managing software reuse, a comprehensive guide to 
strategically reengineering the organization for reusable components. 
Upper Saddle River, New Jersey: Prentice Hall PTR, (1998): 7. 

6 








An important part of validation is the creation of 


formal specifications in the form of assertion statements to 








capture the correct behavior of the software from natural 








language requirements. Assertion statements can be difficult 





to define and produc becaus of natural language 
ambiguities. However, with the use of the reuse concepts, 


libraries can be established. These libraries will contain 





correct assertion statements which have been thoroughly 


tested. The assertion development team can then reuse th 








correct assertion statement and use the accompanying test 


suite to ensure that the chosen assertion matches the 





requirements and proper validation is occurring. 


E. OUTLINE 


This work’s main objective is to facilitate the 


assertion validation process. This is accomplished through 





the use of libraries which contain consistent and accurate 
assertions. We intend to demonstrate that these assertions 
are correct and reusable through the use of testing 


scenarios. These assertions will provide a type of 








engineering control for the IV&V process. 


The organization of the thesis is as follows: 

















Chapter provides background information about IV&V 





and software reuse. The chapter will show a deficiency in 








the current guidance provided for software validation, and 


will describe reuse techniques that have been accomplished 





to date. 

















Chapter discusses the NASA System Reference Model 





(SRM) and the use of assertions in repository libraries. 











Chapter IV discusses the use of patterns to facilitate 


Software reuse. 


Chapter V provides conclusions and recommendations for 


future research. 


II. IV&V AND SOFTWARE REUSE 


A. COMPLEXITY OF SOFTWARE DESIGN 


Anyone who is associated with software design 
understands that software systems can be extremely complex. 


They are SO complex that most softwar ngineering 





researchers often focus their research solely on ways to 


deal with complexity. The reason these systems are so 





complicated can often be traced back to the users’ 
requirements for that system. A system that is required to 
perform several functions will naturally be more complex 


than a system that is required to perform just one function. 





Common problems that occur when developing software include 


failing to match the final product to the customers’ needs, 





or dealing with errors in the software that often reveal 


themselves at the worst possible time and are often costly 








to fix. One way to try to avoid these problems is to 











implement Independent Verification and Validation (IV&V) and 





software reuse into the development of the systems. This 











chapter covers IV&V, how IV&V is conducted, and how software 


engineers are currently leveraging software reuse in 





building software systems. 


B. DEFINITIONS 











Independent: Independence in relation to IV&V is 








defined by the Institute of Electrical and Electronics 








Engineers (LTEEE) using three parameters: technical 
independence, managerial independence, and financial 
independence. 





Technical Independence: “requires the V&V effort to 








utilize personnel who are not involved in the development of 


the software.’”8 





Managerial Independence: “requires that the 








responsibility for the IV&V effort be vested in an 


organization separate from th development and program 





management organizations.”9 








Financial Independence: “requires that control of the 














IV&V budget be vested in an organization independent of the 





development organization.”1° 


Verification: “The process of evaluating a system or 








component to determin whether the products of a given 








deployment phase satisfy the conditions imposed at the start 
of that phase.” 11 Software verification answers the 


question, “Are we building the product right?” 


Validation: “The process of evaluating a system or 





component during or at the end of the development process to 














determin whether it satisfies specified requirements.” 


12software validation answers the question “Are we building 





the right product?” 

















8 Institute of Electrical and Electronics Engineers, Standard for 
Software Verification and Validation, IEEE-STD-1012, June 08, 2005. 

















9 Ibid. 
10 Ibid. 
11 thid. 
12 thid. 
10 





Software IV&V: “a series of technical and management 








activities performed by someone other than the developer of 


a system to improve the quality and reliability of that 





system and to assure that the delivered product satisfies 


the user’s operational needs.”13 


Software Reuse: “the use of existing software artifacts 








in the development of other software artifacts with the goal 





of improving productivity and quality, among other 


factors.”14 


Requirement Specification: “an organization's 





understanding (in writing) of a customer or potential 
client's system requirements and dependencies at a 
particular point in time (usually) prior to any actual 


design or development work.”15 


Pattern: “a body of literature to help software 


developers resolve recurring problems encountered throughout 











all of software development .”1!6 


Cc. IV&V BACKGROUND 


In the early 1940s the first computer was developed to 





calculate artillery firing tables for the United States 








13 R, Lewis, Independent verification & validation: A life cycle 
engineering process for quality software. New York: John Wiley & Sons, 
(1992): xxiii. 


14 Ww. Lim, Managing software reuse, a comprehensive guide to 
strategically reengineering the organization for reusable components. 
Upper Saddle River, New Jersey: Prentice Hall PTR, (1998): 7. 











15 Dp. Le Vie, “Writing software requirements specifications” TECHWR-L 
(MAR 2007) http://www.techwrl.com/techwhirl/magazine/writing/ 
softwarerequirementspecs.htm (accessed July 15, 2008). 





16 p, Appleton, "Patterns and software: essential concepts and 
terminology” CM Crossroads (2000), http://www.cmcrossroads.com/ 
bradapp/docs/patterns-intro.html (accessed September 20, 2008). 


11 








Army's Ballistic Research Laboratory. The design of the 


computer was focused primarily on hardware, not paying much 


























attention to software. In fact, some would say in the early 
stages of computing, software was often ignored. The 
intention of computers at this time was to perform a single 


task. When the task was identified the computers were hard- 





wired to accomplish that task. With the role of software 





being so small the need for IV&V had not yet been 


recognized. However, as time passed and the role and cost of 








software grew, the need for IV&V becam vident. 





In the mid 1940s John Von Neumann came up with two 
concepts that would have a direct impact on software design. 


The first was known as “shared program technique.” “This 





technique states that the actual computer hardware should be 
simple and not need to be hand-wired for each program. 
Rather, complex instructions should be used to control the 
simple hardware, allowing it to be reprogrammed much 


faster.”17 


The second concept he developed was called “conditional 





control transfer.” “This idea gave rise to the notion of 
subroutines, or small blocks of code that could be jumped to 
in any order, instead of a single set of chronologically 
ordered steps for the computer to read. The second part of 


the idea stated that computer code should be able to branch 





out based on logical statements such as “IF” (expression) 


“THEN,” and looped with others such as a “FOR” 





17 C. Robat, “Introduction to software history.” The History of 
Computing Project (October 17, 2006), http://www.thocp.net 
/software/software_reference/introduction_to_software_history.htm 
(accessed June 11, 2008). 
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statement.”!18 The use of these concepts, and others like 
them, allowed software to grow into a more significant part 


of computer design. 


As software grew, so did the cost associated with it. 





In the 1950s, software’s cost was only 20% of the overall 





system cost. In the 1980’s, software costs rose to 80%. 
Today, software costs can be up to 95% of the overall system 


cost.19 These rising costs forced software developers to 





find a way to save money. 





In the late 1950s, one of the leading software 























developers was the Department of Defense (DoD). The DoD 
began to notice projects were consistently behind schedule, 


over budget, and did not provide the required performance. 





This was unacceptable not only for financial reasons but 





because software errors can lead to loss of life, injury, or 








loss of property especially in military systems. The DoD was 





repeatedly surprised by the costly projects because 





“,..softwar development contractors often gave overly 


optimistic assessments of the software development status to 








the DoD.”29 To address this, the DoD launched a plan to 




















conduct IV&V on their software systems in an attempt to get 





accurate assessments of how their projects were doing. The 





18 ¢, Robat, “Introduction to software history.” The History of 
Computing Project (October 17, 2006), http://www.thocp.net 
/software/software_reference/introduction_to_software_history.htm 
(accessed June 11, 2008). 


19 5, Reiss, A practical introduction to software design with C++. 
New York: John Wiley & Sons, 1998, 397-421. 


20 s, Rakin, “Food for thought: What is software quality assurance?” 
Software Quality Consulting (Jan. 2005, Vol.2 No.1), 
http://www.swqual.com/newsletter/vol2/nol/vol2nol.html (accessed June 
O1, 2008). 








first program to use IV&V was the Atlas Missile Program in 





the late 1950s. An independent software tester was hired to 


conduct unbiased testing of the software. 21 








Over time, the role of IV&V continued to develop and in 


the 1970's Mees the U.S. Army sponsored the first 





Significant such IV&V program for the Safeguard Anti- 


Ballistic Missile System.”22 The program was designed to 








identify and eliminate the high risks that are common with 








military systems. It was successful in meeting its goal and 





“By the mid- to late 1970’s, IV&V was rapidly becoming 


popular and in some cases was required by the military 








services...”23 “It was from this effort that IV&V became 





well known within the Department of Defense and _ the 





aerospace communities as an accepted method of ensuring 


better quality, performance, and reliability of critical 





systems.”24 





In the decades following the seventies, IV&V became an 








intricate part of the softwar development process. A 





process that started as “...mostly free-form, not very 





independent, often started too late to be really effective, 


and was sometimes even performed by the very people who were 








developing the system...”29 grew into process where “...a 


21 gs, Rakin, “Food for thought: What is software quality assurance?” 
Software Quality Consulting (Jan. 2005, Vol.2 No.1), 
http://www.swqual.com/newsletter/vol2/nol/vol2nol.html (accessed June 
O01, 2008). 


22 R, Lewis, Independent verification & validation: A life cycle 
engineering process for quality software. New York: John Wiley & Sons, 
1992, xxiii. 

23 Ibid. 

24 Thid. 


25 R, Lewis, Independent verification & validation: A life cycle 
engineering process for quality software. New York: John Wiley & Sons, 
(1992): xxiii. 
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completely independent ntity evaluates the work products 





generated by the team that is designing and/or executing a 





given project...”26 The independent ntity will also 


“,..monitor and evaluate every aspect of the project itself 





from inception to completion.”2’ 





While the cost of conducting IV&V is high, the money 








saved by preventing errors and rework is far greater. In 


1993, the National Aeronautics and Space Administration 





(NASA) established an IV&V facility in the wake of the Space 


Shuttle Challenger accident. The facility was developed as 














part of a plan “to provide the highest achievabl levels of 





safety and cost-effectiveness £Or mission critical 





software.”28 “In 2006, NASA allocated $27 Million to the 
IV&V Facility Budget, of which $19 Million went directly to 











IVéV Services.”29 After conducting a Return on Investment 


analysis, “NASA realized a software rework risk reduction 








benefit of $1.6 Billion in Fiscal Year 2006 alone.”39 From 


the facilities inception at § NASA, it has experienced 








continued growth while providing better software/system 





performance, higher confidence in the software reliability, 


and a reduced maintenance cost. 


26 ¢, Nickolett, “Project due diligence: independent verification and 
validation.” White Paper.Comprehensive Consulting Solutions. Mar 2001: 
1-6. http://www.comp-soln.com/IVV_whitepaper.pdf (accessed June 01, 
2008). 


27 Ibid. 





28 National Aeronautics and Space Administration (NASA), "NASA IV&V 
facility - about IV&V." National Aeronautics and Space Administration 
(NASA), http://www.nasa.gov/centers/ivv/ about/index.html (accessed June 
O01, 2008). 


29 NASA IV&V Facility, “NASA IV&V 2006 annual report.” NASA IV&V 
Facility, http://www.nasa.gov/centers/ivv/pdf/ 
174321main_Annual_Report_06_Final.pdf (accessed June 01, 2008). 
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When performed correctly IV&V can be a crucial part of 





the software development process. The process begins with 








developing Software Integrity Levels (SILs) which “are a 





range of values that represent software complexity, 











criticality, risk, safety level, security level, desired 





performance, reliability, or other project-unique 





characteristics that define the importance of the software 





to the user and acquirer.”3! SILs are then used to determine 
which V&V tasks to perform. The higher the software 


integrity level, the more Vé&éV tasks assigned. SILsS are not 











constant and can change as software evolves to ensure th 








appropriate V&V tasks are being performed. Below is an 





example of SILs based upon the concepts of consequences and 





mitigation potential as well as an example of V&V processes, 














activities, and tasks from the ITEEE Standard for 








Verification and Validation. These examples are provided as 





guidance on how softwar developers can incorporate IV&V 





into their software design to assist in reducing 


specification errors. 














31 Institute of Electrical and Electronic Engineers, Standard for 
Software Verification and Validation, IEEE-STD-1012, June 08, 2005. 
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Description of Software integrity Level 
Software element must execute correctly or grave consequences (loss of life, loss of 


system, economic or social loss) will occur. No mitigation is possible. 


Software element must execute correctly or the intended use (mission) of the system/ 


software will not be realized, causing serious consequences (permanent injury, major 


system degradation, economic or social impact). Partial to complete mitigation is possible. 


Software element must execute correctly or an intended function will not be realized, 


causing minor consequences. Complete mitigation possible. 


Software element must execute correctly or intended function will not be realized, 


causing negligible consequences. Mitigation not required. 





Figure 1. Examples of Software Integrity Levels 
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activities, and tasks. 


V&V processes, 


Figure 2. 
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D. CURRENT GUIDANCE FOR VALIDATION 








Incorporating IV&V into software design is essential to 





reducing specification errors. What softwar ngineers need 











to ensure is when IV&V is applied it is done so correctly. 





The definitions for V&V provided at the beginning of this 
chapter allow for the use of computer-based V&V tools to 


check the correctness of a system or a specific component 





against a formal specification derived from the natural 





language requirements. The specifications are created and 








the final product is then built to satisfy those 


specifications. Validation that is being conducted in 





accordance with the guidelines provided by the IEEE 
evaluates specific components or the final product with the 


specifications. This process is in fact verification. The 




















product is being built correctly according to the 
specifications, however, it is not known ats the 
specifications themselves are correct. It is imperative that 


validation be conducted on the specifications that are 


created to ensure that the requirements for the project are 





understood and that the correct product is built. 


E. SOFTWARE REUSE 


Software reuse iS a practice that began in the 1950s 








with the goal of improving software development productivity 


and quality. For the past twenty years a great deal of 





research has been focused on software reuse and its role in 
software design. Areas that have been given attention 
include but are not limited to reuse libraries, design 
patterns, and reuse using formal specifications of 


requirements. While software reuse holds promise of 
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improving softwar development productivity and software 
quality, the success of reuse is based on the quality of the 
reusable artifacts. The reuse of software that has not been 


verified and validated contradicts the intended goal of 





producing quality software becaus rrors in the software 
may still exist. This reasoning also holds true when 


discussing the use of formal requirements specifications. 





The use of formal requirements specifications is essential 








in the automation of the software verification process. 


However, we assert that the correctness of these formal 





specifications must be first validated before they can be 


used to verify correctness of the software. 





Formal specification has been an active area of 





research for more than two decades. Th requirements 





specification of a software component describes th xpected 


functions and behavior of the software. The ability to reuse 





the software component becomes evident if its structure and 





behavior are compatible with new software being designed. 


Verification has been another popular research topic 








for over 20 years. Automated finite state verification tools 





hav been developed to assist softwar developers in 











verifying system specifications. The users of these tools 








must be capable of specifying the requirements of the system 


they are developing in the specification language the tool 








understands. Behavior for a software component is typically 


specified using temporal logic in an attempt to avoid the 





ambiguity derived from natural language. 
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F. FORMAL SPECIFICATION PATTERNS 


To assist developers in specifying the behavior in a 





temporal logic, Dwyer suggests the use of property 
specification patterns. “A property specification pattern is 


a generalized description of a commonly occurring 





requirement on the permissible state/event sequences in a 
finite state model of a system.”32 These patterns describe 
the essential behavior of a system and provide expressions 
of this behavior in a range of common temporal logics to be 
used with verification tools. The patterns are then given 


distinct names describing their behavior which allows them 








to be mapped to examples of known use, to relationships to 





other patterns, and to specific formalisms. To facilitate 











verification, Dwyer proposes the development of a system of 





property specific patterns for finite state verification 


tools. The system is a set of patterns or library organized 





into one or more hierarchies, with connections between 





related patterns to facilitate the browsing of the system. 








“A user would search for the appropriate pattern to match 





the requirement being specified, use the mapping section to 


obtain the essential structure of the pattern in the 








formalism used by a particular (verification) tool, and then 





instantiate that pattern by plugging in the state formula or 


events specific to the requirement.”33 The use of these 





patterns allows for the specification of critical properties 








32 Mu. Dwyer, G. Avrunin, et.al. “Patterns in property specifications 
for finite-state verification.” Proceedings of the 21st international 
conference on software engineering (1999): 411-420. 

33 Ibid. 
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that exist in software systems and guides users of 


verification tools to express these properties in a 





specification language. 








In 2005, Konrad and Cheng went a step further with 











specification patterns and introduced real-time 
specification patterns that can be used to specify real-time 
properties for embedded systems. Similar to Dwyer’s 








specification patterns, the real-time specification patterns 


contain templates for specifying real-time properties in 








terms of real-time temporal logic.34 This pattern system is 


intended to provide strategies for specifying real-time 





properties in a formal specification language, where the 














properties ar amenabl to automated analysis such as 


verification tools.3° 


Specification patterns and the use of libraries to 





store those patterns provide another form of software reuse. 
This form of reuse aims at reducing the cost and improving 


the quality of formal specification development. However, 











th ffectiveness of the specification pattern reuse depends 
on the correctness and consistency of the resultant 


requirements. Proper validation needs to be performed in 





order to confirm that the requirements are understood. 


Otani et al. explains a concept of developing and 





validating libraries of temporal formal specifications. 








These libraries would include UML Statechart based 


assertions for formal specifications and their associated 





34 B. Cheng and S. Konrad, "Real-time specification patterns." 
Proceedings of the 27th international conference on software engineering 
(2005): 372-381. 


35 thid. 
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validation test  scenarios.3?® We intend to build the 





validation test scenarios with the goal of ensuring that 


specifications within the libraries are indeed error-fr 





and consistent. The following chapter describes the NASA 





System Reference Model (SRM) and its role in capturing a 


modeler’s understanding of a specific problem. 





20> ay Otani, D. Drusinsky, et.al. “Validating UML statechart based 
assertions libraries for improved reliability and assurance.” 
Proceedings of the Second International Conference on Secure System 
Integration and Reliability Improvement (SSIRI 2008), Yokohama, Japan, 
July 14-17, 2008, 47-51. 








23 





TH 


IS PAGE 








NTENT 





ONALLY LEFT BLANK 





24 


IIIT. SYSTEM REFERENCE MODEL 


A. BACKGROUND 


The National Aeronautics and Space Administration 





(NASA) has continuously developed their IV&V program, 











supporting new technologies and better validation and 





verification techniques in an effort to improve the 








validation and verification process. Earlier versions of the 








V&V process included the Criticality and Risk Assessment 








(CARA) and the Software Integrity Level Assessment Process 








(SILAP). Both processes wer found lacking becaus they 





relied on manual examination and independent testing of 


target code. These techniques ar ineffectiv for use in 














validation because there are no links from the requirements 


to the system’s features, capabilities, properties and 








functions. Without formal specifications of the system 





behaviors both CARA and SILAP were unable to validate the 





correctness and completeness of the developer’s 








understanding of the requirements. Finally, the processes 








were unable to locate the subtle errors in increasingly 








complex software-intensiv system. Both CARA and _  SILAP 
evaluated the risk of software components in a system by 


compiling a list of software components and evaluating them 





to prioritize risk assessment, which cannot show that the 





system being built is the correct system. NASA is in the 





process of replacing SILAP with advanced computer-aided 


validation techniques. 
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The NASA IV&V Facility recognized a need for validation 
to be more than a risk assessment; it needed to provide a 


model for the system to show?’: 








e What the system is supposed to do. 








e What the system is not supposed to do and 


e How the system should respond under adverse conditions. 





The NASA IV&V Facility now relies on the use of a 
System Referenc Model (SRM) for each product. “The SRM 





provides the basis for validating the completeness and 





correctness of the targeted requirements set.”38 Once the 








targeted requirements ar developed the independent 





validation team is able to validate those requirements. The 








SRM supports a computer-aided validation technique through 








which th independent validation team’s understanding and 
perception of the problem is validated through the team’s 


representation of the SRM’s features, properties, function, 





and capabilities. It is also during this time that the 
development team is able to discover and correct any 


identified problems or concerns with their understanding of 








the requirements for the intended system. This is important 








because the model holds the responsibility to be complete 





and accurate to serve its intended purpose and_ the 





development team holds the responsibility to ensure that the 





model fulfills that purpose. 





37 K, Woodham, System Reference Model (SRM) development and analysis 
guideline, 1st draft (National Aeronautics and Space Administration 
(NASA), 2007). 


38 thid. 
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B. DEFINITIONS 


The following definitions are described in the SRM 
guideline?9. The definition of dependability will be 
customized to the user’s needs and wants of the system. 


e Dependability: A dependable system is one that provides 
the appropriate levels of correctness and robustness in 
accomplishing its mission while demonstrating the 
appropriate levels of availability, consistency, 
reliability, safety, and recoverability. 

















e Availability: The probability that a system is 
operating correctly and is ready to perform its desired 
functions. 





e Consistency: The property that invariants will always 
hold true in the system. 











® Correctness: A characteristic of a system that 
precisely exhibits predictable behavior at all times as 
defined by the system specifications. 











e Reliability: The property that a system can operate 
continuously without experiencing a failure. 





e Robustness: A characteristic of a system that is 
failure and fault tolerant. 











e Safety: The property of avoiding a catastrophic outcome 
given a system fails to operate correctly. 


e® Recoverability: The ease for which a failed system can 
be restored to operational use. 








Cc. SYSTEM REFERENCE MODEL DEVELOPMENT 





Without a doubt, any process can become overwhelming in 





both cost and time. Thus, it is necessary for the SRM to 
have an appropriate level of specificity so that a 


completion point can be reached. “The appropriate level of 








39 x, Woodham, System Reference Model (SRM) development and analysis 
guideline, 1st draft (National Aeronautics and Space Administration 
(NASA), 2007). 
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V&V is a function of the time available to do the V&V 


evaluations, and this should in turn be a function of the 





risk that will be incurred if the V&V is not done, or the 
risk that will be mitigated if a given level of V&V is 


done.”*9 The SRM still must be developed to a level of 





fidelity to support validation of the system and result in 

















completeness and correctness of the targeted requirements. 





The SRM can b xtremely detailed and can consist of 


high-level use cases, Unified Modeling Language (UML) 





artifacts such as activity diagrams, sequence diagrams and 
object class diagrams, and a set of formal assertions to 


describe precisely the necessary behaviors to satisfy system 





goals, with respect to the three questions stated 





previously. These many artifacts allow the team to properly 


express th requirements through the SRM and ensure that 











their understanding of the requirements is correct. 





Th development of the SRM begins with a scoping 








period. During this time the SRM development team commences 





with a front-end analysis. The front-end analysis ensures 








that the team has a clear perspective of the intended use of 





the model. This high-level abstraction helps the team ensur 








that the model is defined which in-turn drives. the 





objectives of the model development. The scoping period also 


ensures that the SRM development is based on concept-—level 





documentation rather than requirements generated by the 





40 Rp, Logan and C. Nitta, “Verification & validation: process and 
levels leading to qualitative or quantitative validation statements.” 
SAE Transactions vol.113, no.5 (2004), http://bill.cacr.caltech.edu 
/valworkshop/upload/files/UCRLTR-200131lsae04fa.pdf (accessed June 01, 
2008). 
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system developers. Finally, the scoping period should 


finaliz th level of specificity of th requirements so 








that a completion point can be reached. 


The scoping period consists of analyzing the 
constraints, restrictions and targeted tasks and 


requirements to recognize the depth of the modeling needed. 





Additionally, requirements that will not be modeled in the 





SRM are identified and th team nsures that sufficient 








concept documentation is available to continue. The concept 
documents used during the process are found in many forms of 
stakeholder inputs from mission statements to concepts of 
operations. The scoping period ends with a clear 
understanding of the system elements that need to be 


addressed and the depth that they need to be defined. The 





level of fidelity should be determined at this point to 
ensure completeness and correctness of the targeted system 


requirements. 


The next stages of the SRM development are accomplished 


through the development of use cases and UML artifacts as 





well as supporting assertions. The SRM team, using the 





conceptual documents, will begin by documenting system 








behaviors. It is during this time that the system goals 








should be identified and a traceability matrix developed, 





populated with these top-level goals. Additionally, the 





operational environment must be identified and the 


traceability matrix should be populated with operation 








environment characteristics that need to be addressed by the 








system model. The top-level use cases developed to address 








the overall system goals are peer reviewed to validate that 





the preliminary use case set spans the high-level 
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description of the system and is documented in the 








traceability matrix. The top-level use cases are abstracted 


from the details of the system and are goal-oriented. These 








use cases help the developers to get a clear understanding 


of the process and problems to be solved as well as the 





goals and objectives of the system. The top-level use cases 





then are refined into lower-level use cases and activity 
diagrams which can be mapped to sequence diagrams. The 


process continues to become more specific to ensure that the 





goals and objectives are accomplished but also to verify 


that their constraints are also adequately captured. The 





diagrams should provide a complet representation of the 


behavior expected to be displayed by the system. 





Additionally, all behaviors should be mapped and defined 





into the traceability matrix and peered reviewed to ensure 


correctness. The overall goal is to ensure that the top- 








level use cases have been refined into detailed lower-level 











uses cases that represent not only the Main Success Scenario 


(MSS) but are fully elaborated to nsur necessary 








extensions are also represented. Finally, the modeling team 








has to ensure that any dependability considerations are 
addressed and represented in the model. This entire effort 
should represent the desired system behaviors as well as any 


necessary extensions and assertions that map to the top- 





level goals and requirements. The model is ready to be 


validated. 


D. VALIDATING THE SYSTEM REFERENCE MODEL 


The newly developed SRM is a representation created by 





the SRM development team and is a result of the teams own 


perceptions and understanding of the desired system 
30 


behaviors. As such the representation could be wrong if the 


team misinterpreted th desired behaviors of the system. 





This is why the SRM must be validated to reduce 
specification error as well as to ensure that the behavior 
requirements created by the SRM development team are 
measured against the SRM for correctness. The SRM is a model 
of the intended system and it must meet any dependable 


considerations in order for the intended system to be so as 





well. 


The validation process is twofold and can begin with a 
formal review and tracing of the UML artifacts to include: 
use case definitions and models, supporting assertions, and 


activity diagrams. Other artifacts reviewed include the 





complete set of system-behavior definitions based on 


stakeholder goals and system constraints and operations 





environments defined in the concept documentation. During 


this review the formal tracing of the requirements from the 








top-level to the more refined lower-levels and the activity 








diagrams and sequence diagrams helps to identify the 


subsystems and components responsible for the system 





requirements. Additionally, during this process all the 








requirements are elicited and peer reviewed. This ensures 








that all targeted requirements hav been identified and 








traced through the artifacts. During this time all necessary 








objects and events are labeled, identified, and checked to 





nsure there are no unnecessary objects or events. The above 


process nsures that all targeted requirements are fully 











detailed in accordance with their goals. During each review 





each step is subjected to extensive group review to validate 


that the SRM is a complete and unambiguous representation of 





the system. 
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The second step of the validation process is to execute 








as much of the model as possible through computer-aided 


auditing. Run-time verification of formal assertions is able 





to check for inconsistency, omission and errors in the SRM. 


By executing as much of the model as possible it increases 





th vidence that the model being developed is the correct 








system. The independent validation team is able to use the 





evidence of validation to ensure that the SRM is the correct 


system. 





The IV&V team’s requirements elicitation and validation 





tasks produc deliverabl packages, consisting of: UML 





models for reference model constituents, natural language 


assertions, formal representation of the assertions, and a 





validation test suite for each assertion. The test suites 





are detailed and include tests that cover multiple scenarios 














that meet the requirements of the assertions, and will be 


discussed further in the next chapter. Thes deliverabl 





packages are th vidence gathered to decrease specification 








errors and must be done to validate the SRM and provide 





evidence of dependability of the system. 


The SRM is intricate and detailed in order to show its 


dependability. But before dependability can be shown the SRM 











assertions must be validated to decreas specification 
errors. In fact the assertions should precisely model the 
required behavior of the system and if they are able to do 








so the model is on its way to being validated. But 
assertions also have to be validated and are validated 
through an execution-based model checker for dependability 


of the model under test. 
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E. INCREASING THE USABILITY OF THE SRM 





The difficulty with assertions is in their creation. It 














not only takes time and effort, but the correctness of the 


executable assertions depends on the ability of the modelers 





to specify correct assertions. It is difficult to specify 





and develop correct assertions. The modelers must have a 





correct representation of the structure and behavior of the 








SRM, the assertions must also be correct. If faulty 





assertions ar used they are not effective in the IV&V 





process. We believe that a library built with correct 





assertions would nabl th assertions to be reused. This 


could both decrease the burden on the modelers to develop 











the assertions and improve the ability of the independent 





validation teams to validat th dependability of the 


software. 


33 





TH 


IS PAGE 








NTENT 





ONALLY LEFT BLANK 





34 


IV. BUILDING AN ASSERTION LIBRARY 


A. BACKGROUND 


As mentioned in the previous chapters the SRM is a 





representation created by th developers as a result of 


their own understanding of the desired system behaviors. The 





SRM must be validated to reduce specification errors. One of 


the ways to do this is through assertions which precisely 





model th required behavior of the system and are the 





foundation of the SRM. Through testing and modeling 


assertions the independent validation team begins to 





comprehend the problem domain and refine any problems to 





ensure that the SRM meets the user’s requirements and the 
correct system is built. The current way to build assertions 
is to develop the assertions from natural a language 


description of the user’s understanding anew every time; 





this can be a time-consuming and error-prone undertaking. We 





believe that an assertion library can help ease these tasks 





by providing validated assertions which can be reused. 


The purpose of this chapter was to construct an 





approach to building an assertion library with a small 


number of assertions that have been validated for 





correctness and are reusable. We define a library to be a 


collection of assertions that are stored, collectively 





shared and can be filled with more assertions as needed. The 


assertions in the library are validated through the use of 





test scenarios that we designed. The test scenarios are 








patterns which test the assertion for the required behavior. 


The purpose of the test suites is to disambiguate the 


3:5 


assertions, and test for correctness meaning that the 





assertions accurately reflect the natural language statement 





as we intended. 


The assertion library would be built so that the 








assertions are reusable and adaptable for future projects. 


Softwar developers can select from the library any 








assertions that meet their needs and adapt them or use them 














as an example to build their own. In each case w nsured 


that the assertion was general to increase the ability to be 








reused as well as be more relevant to the library. We hope 








that through this process that software developers will be 





able to use our correct assertions in the library for their 








own use and reuse, lessening their burden and reducing 


specification errors in the software. 





B. STATECHART ASSERTIONS 


The libraries are built through the use of “UML 


statechart based temporal assertions for formal 





specifications.”4! The UML statecharts are developed from 





both the research efforts of Harel, who first proposed the 


use of statechart diagrams as a visual approach to modeling 





the behavior of complex reactive system, and Drusinsky who 








both increased and extended the use of statechart diagrams 





to specify formal assertions. Drusinsky was able to extend 


the use of statecharts as formal assertions for temporal 








behavior with “the inclusion of a built-in Boolean flag 
bSuccess and a corresponding isSuccess method which 


specifies the Boolean status of the assertion true if the 








41D, Harel. Statecharts: A visual approach for complex systems, 
Science of Computer Programming, vol.8, no.3. (1987): 231-274. 
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assertion succeeds and false otherwise.”44 The statechart 





assertion indicates that “formalism is supported by 
StateRover, a design entry, code generation, and visual 


debug animation tool for UML statecharts combined with 





flowcharts.”43 Assertion statecharts can be nondeterministic 





and deterministic depending on the needs and wants of the 





developer and modeler. For example, the developer might want 








a nondetermistic statechart if there are nested requirements 


which can be more difficult to write and less readable in a 





deterministic solution. Or alternatively if the assertion 


needs to be active in runtime, then a deterministic 








statechart might be a better solution because of the 


overhead incurred in the nondeterministic statechart at 





runtime. 


Finally, it is important to understand the proper use 





of a statechart assertion. Remember that the assertion uses 





the “built-in Boolean variable name bSuccess, and a 


corresponding method called isSuccess(), both automatically 





created by the code generator”44 to make a statement about 
the assertion’s correctness. The default settings of the 


assertion statechart variable bSuccess is set to true. To 








appropriately test success and failure, the modeler needs to 


42> Dy, Drusinsky. Modeling and verification using UML statecharts - a 
working guide to reactive system design, runtime monitoring and 
execution-based model checking. Elsevier Inc., 2006. 














43 p, Drusinsky, M. Shing, K. Demir, “Creation and validation of 
mbedded assertion statecharts”, Proc. 15% IEEE International Workshop 
in Rapid System Prototyping, Greece (June 14-16, 2006): 17-23 














44 p, Drusinsky. Modeling and verification using UML statecharts - a 
working guide to reactive system design, runtime monitoring and 
execution-based model checking. Elsevier Inc., 2006. 
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ensure that the assertion enters the error state and the on- 





entry action assigns bSuccess=false when the assertion 


fails. 
Cc. ASSERTION VALIDATION 

Once the natural language has been translated into an 
assertion the assertion must then be validated. The 


assumptions in the statechart must be tested to ensure that 


the statechart assertion correctly represents the intended 





behavior the modeler has in mind. We need to run validation 





test scenarios against the statechart assertion. 








In each case the validation test suite resolved the 





ambiguities of the natural language specification. The tests 





were meaningful in that they nsured each assertion were 


distinguishable from each other. The assertions were tested 











and we did find that, when w tested them, we had to 
disambiguate the natural language ourselves to ensure that 


we truly understood what we were describing. 


The two kinds of errors that are commonly found were 

















“implementation errors resulting from mistakes in the 
statechart assertion, and errors or ambiguities in the 
natural language statement.” 4° In the first case, the 
statechart behavior does not match the modeler’s intended 





behavior. The second case was more difficult because it 





depended how we as individuals understood the natural 








language statement and how we as individuals clarified the 





45 7, Otani, D. Drusinsky, et.al. “Validating UML statechart based 
assertions libraries for improved reliability and assurance.” 
Proceedings of the Second International Conference on Secure System 
Integration and Reliability Improvement (SSIRI 2008), Yokohama, Japan 
(July 14-17, 2008): 47-51. 


45 Ibid. 
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ambiguities. It was by running the test scenarios that we 





were able to identify these errors and modify our 
assumptions and assertions accordingly in order to correct 


the assertions. 


Otani et al.*® revealed that there are some types of 














patterns that must be part of every validation test-suite: 








e® Obvious success. We expect that the statechart 
assertion being validated to succeed on such a test. 






























































e Obvious failure. W xpect that the statechart 
assertion being validated to be violated on such a 
test. 

e Event repetitions. We creat vent repetitions and 
assure that the assertion, if applicable, is not 
written in a manner that only observes the first 
occurrence of a triggering event P ina sequence 
OF “Ps S'3 

e Multiple time intervals. If the assertion requires 
it, we check that it handles multiple time intervals 
or scenarios. By using this validation test 
pattern we assure that an assertion is not written 
in a manner that observes only a single time 
interval. 








e® Overlapping time intervals. If the assertion 
requires it, we check that the assertion can handle 
overlapping time intervals within a scenario. 











Once the types of patterns were clarified w then 





designed our test suite to adhere to the above categories, 





combining them if suitable and ensured that there were an 





appropriate number of tests per test suite that would 





validate the assertion. 





sae Otani, D. Drusinsky, et.al. “Validating UML Statechart Based 
Assertions Libraries for Improved Reliability and Assurance.” 
Proceedings of the Second International Conference on Secure System 
Integration and Reliability Improvement (SSIRI 2008), Yokohama, Japan, 
(July 14-17, 2008): 47-51. 
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D. ASSERTION SCENARIOS 


The first assertion statement that we described is: 
Whenever P then Q within T. The assertion statechart (shown 


in Figure 3) was diagramed as follows: 





Orit ~|-———____» OP 


On-Entry/timer.restart(); 





0 
| \, OLY 
N, Oy 
% sa timeoutFire[]! 
oO Error 
On-Entry/bSuccess = false; 
Figure 3. Whenever P then Q within T 


Our interpretation of the assertion statement above is: 





if P occurs (timer is reset at every P) then the event Q 





will eventually occur within the time interval. The built in 





event, timeoutFire(), fires after 30 sec. In case of a P 
repetition before a Q the 30 second duration will be 


measured from the first p. We used the following patterns to 
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correctly disambiguate the natural language and ensure that 





the assertion statement accurately reflects the natural 





language as we identified and desired. As described earlier 








in the chapter we covered all appropriate testing patterns. 


Obvious success: 


P Q 





P; incerTime(25); QO; incrTime(6) (timeout has occurred). 





We expected this test to be a success. Our expectation was 


confirmed. 


Q 








Q; incrTime (31) (timeout has occurred). We expected 





this test to be a success becaus W ar testing for 
violations of the assertion and Q by itself does not violate 


the assertion. Our expectation was confirmed. 


Obvious failure: 





P; incrTime (31) (timeout has occurred). We expected 








this test to fail because it did not meet the constraints of 





the assertion. Our expectation was confirmed. 
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Overlapping time intervals: 





In this test and the next test we ensure that the 


assertion observes more than the first P in a sequence of 


P's. 


P PQ 





Ps incrTime (15); PB; incrTime (5); Q; incrTime (26) 


(timeout has occurred). Our goal in this test was to ensure 





that the assertion could handle overlapping time intervals. 





We expected success. Our expectation was confirmed. 


P QP Q 





P; incrTime (5); Q; P; incrTime (31) (timeout has 





occurred); Q. Our goal in this test was to test overlapping 











time intervals for an expected failure as this test does not 





meet the constraints of the assertion. Our expectation was 


confirmed. 





Multiple Intervals: 





We tested for multiple intervals in this test and the 








next to ensure that the assertion would observe more than a 


single time interval. 
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P; ancrTime(10); QO; incrTime(20); P; incrTime(10); OQ; 


incrTime (21) (timeout has occurred). We expected success 








because it meets the requirements of the assertion. We set 


bSuccess = true. Our expectation was confirmed. 


P Q P 














P; incrTime (10); Oe incrTime (20); P; incrTime (31) 
(timeout has occurred). We tested for multiple intervals 
expecting failur becaus of the constraints of the 








assertion. Our expectation was confirmed. 





The second core assertion statement that we described 
was: Whenever P then no Q within T. The assertion statechart 


(as shown in Figure 4) was diagramed as follows: 
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© Ny PLY 


Onnit ‘. PLY OP “ 





























——__—_»> 
On-Entry/timer.restart(); 
My timeoutFire[]! 
\, OL 
ee oll O Error 
On-Entry/bSuccess = false; 
Figure 4. Whenever P then no Q within T 


Our interpretation of the assertion is if P, then 





within the time interval for P no Q will appear. The built 
in event, timeoutFire(), fires after 30 sec. A P repetition 
would reset the timer. We used the following patterns to 


disambiguate the assertion statement. 


Obvious success: 
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P; incrTime (31) (timeout has occurred). We expected 








this test to be a success because no Q occurred which meets 





th requirements of our assertion. Our expectation was 





confirmed. 


Q 





Q; incrTime (31) (timeout has occurred). We expected 








this test to be a success becaus W ar testing for 
violations of the assertion and Q by itself does not violate 


the assertion. Our expectation was confirmed. 


Obvious failure: 


P Q 





P; incrTime (25); QO; incrTime(6) (timeout has occurred). 


This test was expected to be a failure because it violates 





th requirements of the assertion. Our expectation was 


confirmed. 


Overlapping time intervals: 








In this test and the next test we ensure that the 
assertion observes more than the first P in a sequence of 


Ps. 


45 





Pe incrTime(15); P; incrTime (S92 OF incrTime (26) 





(timeout has occurred). Our goal in this test was to ensure 


that the assertion could handle overlapping time intervals. 





The test was expected to be a failure. Our expectation was 


confirmed. 





P; incrTime(10); P; 





incrTime (31) (timeout has occurred); 


Q. The test was expected to be a success becaus 





QO was not 


injected during the P intervals. Our expectation was 
confirmed. 
Multiple Intervals: 

















We tested for multiple intervals in this test to ensure 





that the assertion would observe more than a single time 


interval. 





P; incrTime (30); Py incrTime (15); Q; incrTime (16) 
(timeout has occurred). This test was expected to be a 
failure because it does not meet the constraints of the 


assertion. Our expectation was confirmed. 
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E. CONCLUSION 


When we first defined the assertions in natural 
language we discovered that almost all assertions can be 
ambiguous and difficult to define at first. The natural 


language statements meant different things to different 





people. “Tf P then Q within T” could mean an interval T 
measured from the first or the last P depending on how it 
was defined and what the software developers wants to test. 
We disambiguated each assertion according to the most 
general and useful definition; this meant that in most cases 
the assertion would be general and not specific so as to be 


more useful. There was additional difficulty as can be 





expected with any new system as StateRover is still in 
development. But we were able to succeed after several 


restarts and debugging help. Finally, during our 





disambiguation period we fell victim to the statechart 








default which is bSuccess = true. During the testing period 
we expected one result and received something completely 


different. This led us to additional testing and 





clarification of the assertions and we had to ensur very 





time that the assertion test was not successful because the 


bSuccess flag was set to true, but rather because the test 





was actually correct. 





This process is incredibly interesting and reguires 


clarity of thinking as well as the ability to break down 











natural language. It is not simple but the process invokes 
greater understanding of the validation process and the 
validity of the assertion library. We feel that these 


assertion statements can be built upon and reused for the 








benefit of validation purposes. 
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Additional assertion statecharts and validation test 
suites that we defined and tested can be found in Appendix 
A. A final assertion statechart and validation test suite 
that has merit but is not as valuable as previous mentioned 


assertions can be found in Appendix B. 
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V. CONCLUSION 


A. SUMMARY AND CONTRIBUTIONS 


Software has become a vital part of our everyday lives. 


Whether we refer to our military systems, medical systems, 








or our financial systems, software is a part of them and has 





become something that we now depend on. In our thesis we 





concentrate on requirements and their formal specification, 


and we discuss a method to reduce specification errors. We 





strive to find a better technique to answer the question 
“Are we building the right product?” Validation presents a 


means of assuring that software satisfies the user’s 





requirements. It is viewed as a way of saving time and money 
that could otherwise be wasted if a product design is not 


built correctly and rework needs to be conducted. A problem 








that can exist when conducting validation is not conducting 
validation early enough in the design process. Often the 
user’s requirements are reviewed and specifications are 


developed. The product design is then built according to the 





specifications. Once the product design is built validation 


is conducted by comparing the resultant product with the 





original requirements. As software partition of systems 





continues to grow and become more complex we assert that 





validating a product after it is developed is too late in 
the process. At that stage the amount of time and cost of 
rework that may need to be performed is too large. 


Validation needs to begin earlier in the design process by 








ensuring the specifications themselves are correct and 





consistent. 
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At present several ways to conduct validation exist. 
Some guidance that is provided actually describes 


verification when referring to validation by having the 





product design compared to the specifications for the 








project. Others suggest what we have already mentioned and 


that is to check the final product against the user 





requirements. To conduct validation we introduced a process 
of developing and validating temporal formal specifications 


in the form of statechart assertions. Included in our work 














are validation test scenarios intended to ensure 





specifications are in fact correct prior to moving forward 





with a project. The goal is to make available multiple 





libraries of pre-vetted assertions to facilitate validation. 








This research described the attributes of IV&V as well 
as software reuse and explores a concept of combining the 


process of validation and reuse in an attempt to yield a 





repeatable validation technique. Sampl requirements were 
identified and then formal specifications in the form of 
statechart assertions were created to capture the 


requirements. Testing scenarios wer then developed to 








determine if the statechart assertions were accurate and 








consistent with the original requirements. Once these 





assertions are proven to be accurate they can be stored in a 








library for future reuse. Our intentions are to ensure that 


specifications used to build a product are validated prior 





to time and money going into building the final product. By 








using an assertion repository filled with correct assertions 





to build the specifications for a design, the engineer can 


be sure that the specifications used to build the final 





product are correct. sa errors are found in the 


specifications the engineer can go back and find out where 
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the error is coming from. This would be faster and cheaper 





than correcting software that has already been developed in 





accordance with incorrect specifications. 


B. FUTURE WORK 








The goal in both the DoD and the software industry is 





to produce software that is cost ffective, reliable, 
maintainable, and above all usable. The current guidance on 
verification and validation that exists does not provide a 


technique to show engineers how to create software that 








possesses these attributes. The guidance that does exist 
leaves software engineers to develop their own verification 


and validation methods. 


The amount of work that could be conducted in the 


software industry to ensure reliable software is being 





produced is abundant. We have established an approach for 


developing and validating statechart assertions as a road 








map to produce reliable software. One avenue of future work 
would be to further expand this approach by developing 
additional assertions that apply to a specific domain. For 


example, select a domain of interest such as theatre 











ballistic missile defense. Then, determine requirements that 








exist within that domain. Onc th requirements for the 





specific domain are understood, translate the natural 





language of the requirements into assertions as we did in 





chapter IV. Then validate the assertions through the use of 





test cases to ensure that the statechart assertions 





correctly represent the intended behavior the modeler has in 





mind. 
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Another avenue of future work would be to create a 


library to store the assertions in. When creating the 








library the developer will need to consider the size of the 





library and how many assertions will be placed in it. The 


developer will need to decide if several libraries are to be 








developed to categorize the different assertions or if the 
assertions will be organized within one library in a manner 
that will be easy to search. Once the organization of the 


library is decided information retrieval will need to be 








focused on. How will assertions be retrieved or called from 
the library? What will be the best interface to facilitate 


information retrieval and the use of the assertions? The 





goal should be to find an acceptable interface that does not 


cause errors of its own. Another area to look at is the 





adaptation of the assertions to a library environment. Do 
they perform as expected? One goal the developer should seek 


is to automate the processes of organizing, retrieving 





information from, and interfacing the libraries as much as 


possible in an attempt to reduce errors. 


Finally, once a library is developed, a future project 





could focus on how to best maintain that library to 























facilitate future use. One item to consider is if certain 
assertions are used more frequently than others. In this 
case the developer would want to set up the library in a way 





that the frequently used assertions can be searched befor 





the rest of the library is searched. A way to enable this 





would be to maintain a count of how often each assertion is 


used. Also if an assertion is proven not to be used, a way 





to comment the assertion out in the library to eliminate it 





from future searches may prove useful. Doing this may be a 


way to enable faster searches thereby saving time in the 
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development process. By commenting the assertion out rather 


than removing it from the library it can still be included 





in future searches if it is decided that it is needed. 
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APPENDIX: ADDITIONAL ASSERTION DIAGRAMS AND TEST 
SUITES 


A. ADDITIONAL ASSERTION DIAGRAMS BOUNDED BY TIME 


1. Whenever P Then Less Than N Qs Within T 
oe 4 oF a 
——_—_> On-Entry/timer.restat(}... 


On-Entry/bSuccess = false; 


Ont S, oty as 
Ores Oe 
6 UY 


On-Entry/cnt ++; 


\, timeoutFire[ ‘ 
 timeoutFire[]’ 


cnt <N 







\, [True]! 


ntN = 2; 

ntcnt; 

TRTimeoutSimulatedTime timer = new 

TR TimeoutSimulatedTime(30, ths); 
/{timeoutFire (transition) 


\, [False]/ 





Whenever P then less thanN Q's within T 





P; incrTime(31) (timeout has occurred). We expected an 





obvious success. Our expectation was confirmed. 





P; incrTime(5); Q; incrTime(26) (timeout has occurred). 


W xpected an obvious success. Our expectation was 





confirmed. 
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P; incrTime (5); QO; incrTime (5); QO; incrTime (21) 





(timeout has occurred). We expected failure since we set N 


to 2. Our expectation was confirmed. 


PQQ Q 





P; incerTime(5); QO; ancrTime(5); QO; incrTime(5); OQ 


. 
y 


incrTime(16) (timeout has occurred). We 





expected failure 


since we set N to 2. Our expectation was confirmed. 





P; incrTime(5); Q; 


incrTime (26) (timeout has occurred); 





P; incrTime(5); QO; incrTime(5) QO; incrTime(21) (timeout has 





occurred). We tested for multiple intervals 








in this test to 
ensure that the assertion would observe more than a single 


time interval. 





We expected failure in the second interval 


since we set N to 2. Our expectation was confirmed. 
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2. Whenever P Then Less Than or Equal to N Qs Within 
T 


\ Or 
@_.0 Init 3 PLY = Obrer 
> On-Entry/timer.restat(} ... 


On-Entry/bSuccess = false; 


~~ Ont \, ory ane 
Ores ue: 
. OLY 


On-Entry/cnt ++; 


\, timeoutFire[ ra 
4, timeoutFire[]’ 
N 


cnt <= 







\, [True} 


ntN= 2; 

ntcnt; 

TRTimeoutSimulatedTime timer = new 

TR TimeoutSimulatedTime(30, thé); 
i}timeoutFire (transition) 


\, [Fabel] 





Whenever P thenless than or equal to NQ's within T 





P; incrTime(31) (timeout has occurred). We expected an 





obvious success. Our expectation was confirmed. 


P Q 





P; incrTime(5); Q; incrTime(26) (timeout has occurred). 


W xpected an obvious success. Our expectation was 





confirmed. 





Sf 


P; incrTime (5); Q; incrTime (5); QO; incrTime (21) 





(timeout has occurred). We expected an obvious success. Our 





expectation was confirmed. 


PQQ Q 





P; incerTime(5); QO; incerTime(5); QO; ancrTime(5); OQ ; 


incrTime(16) (timeout has occurred). We 





expected obvious 





failure since we set N to 2. Our expectation was confirmed. 


P Q PQQQ 





P; incrTime(5); Q; 





incrTime (26) (timeout has occurred); 


P; incrTime (5); OQ: incrTime (5) Q; incrTime (5); Q; 


incrTime(16) (timeout has occurred). We tested for multiple 








intervals in this test 





to ensure that the assertion would 


observe more than a single time interval. We expected 





failure in the second interval since we set N to 2. Our 





expectation was confirmed. 
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3% Whenever P Then Equal to N Qs Within T 


Pp 
@ > Oni “s PLY O Oetror 
PTT TES a On-Entry/timer restat(}... —S 
O ™ \, ay (e Count_O On-Entry/bSuccess = false; 
N, [Truey! \ N, OLY 


On-Entry/cnt ++; 
‘ = A timeoutFire[}’ 


cnt ==N 






[False ]/ 
TRTimeoutSimulatedTime timer = new 


TR TimeoutSimulatedTime(30, this); 
i{timeoutFire (transition) 


Whenever P then equal to NQ's within T 





P; incrTime (31) (timeout has occurred). We expected 





obvious failure since we set N to 2. Our expectation was 


confirmed. 


P Q 





P; incrTime(5); Q; incrTime(26) (timeout has occurred). 
We expected obvious failure since we set N to 2. Our 


expectation was confirmed. 
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P; incrTime (5); Q; incrTime (5); 


Q; incrTime (21) 





(timeout has occurred). We expected obvious success since we 


set N to 2. Our expectation was confirmed. 


PQQ Q 





P; incerTime(5); QO; incrTime(5); OQ; 


incrTime(16) (timeout has occurred). We 








incrTime (5); 


Q ; 


expected obvious 


failure since we set N to 2. Our expectation was confirmed. 


PQQ PQQQ 





P; incrTime(5); Q; incrTime(5); QO; incrTime (21) (timeout 





has occurred) ; P; incrTime (5); QO; 


incrTime (5) 


incrTime(5); QO; incrTime(16) (timeout has occurred). 











the assertion would observe more than 





interval. W xpected failure. Our 


confirmed. 
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tested for multiple intervals in this test to ensure 


a single 


expectation 


Q; 
We 


that 





time 


was 


4. Whenever P Then Greater Than or Equal to N Qs 





Within T 
\, timeoutFire[]’ 
Onit Op Oper 
e- Se aaa 
—_—_ On-Entry/timer.restat(}... On-Entry/bSuccess = false; 
> Oni 
N, Of]fent ~ | \, [Fabel 
cnt >=N 


ntN=2; 

int cnt =O; 

TRTimeoutSimulatedTime timer = new 
TR TimeoutSimulatedTime(30, this); 
{}timeoutFire (transition) 


twhenever P then there must be greater than or equal to NQ's within T 








P; increment time to 31; timeout. We expected obvious 


failure since we set N to 2. Our expectation was confirmed. 


P Q 





P; incrTime(5); Q; incrTime(26) (timeout has occurred). 


We expected failure since we set N to 2. Our expectation was 


confirmed. 


P Q Q 
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P; incrTime (5); QO; incrTime (5); QO; incrTime (21) 





(timeout has occurred). We expected success since we set N 


to 2. Our expectation was confirmed. 


PQQ Q 





P; incerTime(5); QO; <ancrTime(5); QO; incrTime(5); OQ 


° 
y 


incrTime(16) (timeout has occurred). We expected success 





since we set N to 2. Our expectation was confirmed. 


PQQPQ 





P; incrTime (5); QO; incrTime (5); QO; incrTime (5) ise 


incrTime (5); QO; <incrTime(26) (timeout has occurred). Our 





goal in this test was to ensure that the assertion could 


handle overlapping time intervals and that the assertion 





observes more than the first P in a sequence of Ps. The test 





was expected to be a failure since we set N to 2 which 


violates the second P. Our expectation was confirmed. 


PQQ PQ 





P; incrTime(5); Q; incrTime(5); QO; incrTime (21) (timeout 





has occurred); P; incrTime(5); QO; incrTime(26) (timeout has 





occurred). We tested for multiple intervals in this test to 





ensure that the assertion would observe more than a single 








time interval. We expected failure since we set N to 2. Our 





expectation was confirmed. 
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5. Whenever P Then Greater Than N Qs Within T 









ntN= 2; 
ntcnt; 
TRTimeoutSimulatedTime timer = new 
TRTimeoutSimulatedTime(30, ths); Ccount_o \, timeoutrrely 
Error 
Init On-Entry/timer.restat(y... O 
pt) ¥, 
@ Ont > On-Entry/bSuccess = false; 
\, PLY 
N, Qflfcnt +4; VS [False]/ 
Se [True}’ cnt > N 
Whenever P then Greater thanN Q's within T 
P; incrTime (31) (timeout has occurred). We expected 





failure since we set N to 2. Our expectation was confirmed. 


P Q 





P; incrTime(5); Q; incrTime(26) (timeout has occurred). 
We expected failure since we set N to 2. Our expectation was 


confirmed. 
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P; incrTime (5); QO; incrTime (5); QO; incrTime (21) 





(timeout has occurred). We expected failure since we set N 


to 2. Our expectation was confirmed. 


PQQ Q 





P; ainerTime(5); OQ; incrTime(5); OQ; incrTime(5); Q; 





incrTime (31) (timeout has occurred). We expected success 


since we set N to 2. Our expectation was confirmed. 


PQQQP Q 





Py incrTime(5); Q; incrTime(5); Q; incrTime (5) QO; 


incrTime (5); P; incrTime(10); OQ; incrTime(21) (timeout has 





occurred). Our goal in this test was to ensure that the 


assertion could handle overlapping time intervals and that 





the assertion observes more than the first P in a sequence 


of P’s. The test was expected to be a failure since we set N 





to 2. Our expectation was confirmed. 
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P; auncerTime(5); QO; j<ancerTime(5); Q; %$incerTime(5); Q; 


incrTime(16) (timeout has occurred); P; incrTime (5); QO; 





incrTime(26) (timeout has occurred). We tested for multiple 





intervals in this test to ensure that the assertion would 


observe more than a single time interval. We expected 





failure since we set N to 2. Our expectation was confirmed. 


6. Whenever P Then Q and R Within T 





Oc 


‘ | 
atic | \. Oty 


Op i \,, timeoutFire[] 


O Init N, PL Orr 
On-Entry/timer.restart()}; 
- > par yh 0 
> On-Entry/bSuccess = false; 


\,, timeoutFire[]/ 


1 1 \,, timeoutFire[]/ 


\ oly Na REY Or 








N. OF 
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P; incrTime (31) (timeout has occurred). We expected 





failure. Our expectation was confirmed. 


Q 





Q; incrTime(31) (timeout has occurred). We expected 





success. Our expectation was confirmed. 


R 





R; incrTime (31) (timeout has occurred). We expected 





success. Our expectation was confirmed. 





P; incrTime(5); Q; incrTime(26) (timeout has occurred). 


We expected failure. Our expectation was confirmed. 


66 





P; incrTime(5); R; incrTime(26) (timeout has occurred). 


We expected failure. Our expectation was confirmed. 


P Q R 





P; incrTime(5); Q; incrTime(5); R; incrTime (21) (timeout 





has occurred). We expected success. Our expectation was 


confirmed. 


P R Q 





P; incrTime(5); R; incrTime(5); QO; incrTime (21) (timeout 





has occurred). We expected success. Our expectation was 


confirmed. 


P PQR 





P; aincerTime(5);P; incrTime(5); OQ; incrTime (5); _ R; 


incrTime (26) (timeout has occurred). Our goal in this test 





and the next was to ensure that the assertion could handle 








overlapping time intervals and that the assertion observes 
more than the first P in a seguence of Ps. The test was 


expected to be a success. Our expectation was confirmed. 


67 





P; aincerTime(5);Q; aincerTime(10) P; incrTime (5); BR; 


incrTime (26) (timeout has occurred). The test was expected to 





be a failure. Our expectation was confirmed. 


PQR PQ 





P; incrTime(5); Q; incrTime(5); R; incrTime (21) (timeout 








has occurred); P; incrTime(5); QO; incrTime(26) (timeout has 
occurred). W xpected failure. Our expectation was 
confirmed. 





In this assertion statement we did not differentiate 
between Q or R coming first, our intention was to ensure 
that the combination of both Q and R regardless of order 


resulted in a successful test. 
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7. Whenever P Then Q or R Within T 


\, oy 
\, OY 


O Int |< O P O Error 


@- On-Entry/timer.restart0; | \. timeoutFire[} | On-Entry/bSuccess = false; 
ha seeeeee Ny 











P; incerTime(31) (timeout has occurred). We expected 


failure. Our expectation was confirmed. 








Q; incerTime(31) (timeout has occurred). We expected 


success. Our expectation was confirmed. 
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R; incrTime (31) (timeout has occurred). We expected 


success. Our expectation was confirmed. 


P Q 






30 


P; incrTime (5); Q; incrTime(26) (timeout has occurred). 


We expected success. Our expectation was confirmed. 


P R 





P; incrTime(5); R; incrTime(26) (timeout has occurred). 
We expected success. Our expectation was confirmed. 


P Q R 





P; incrTime(5); Q; incrTime(5); R; incrTime (21) (timeout 





has occurred). We expected success. Our expectation was 


confirmed. 
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P; incrTime(5); R; incrTime(5); QO; incrTime (21) (timeout 





has occurred). We expected success. Our expectation was 


confirmed. 





P; incrTime (5);Q; incrTime(15); P; incrTime(5); R; 





incrTime (26) (timeout has occurred). Our goal in this test 


and the next was to ensure that the assertion could handle 





overlapping time intervals and that the assertion observes 





more than the first P in a sequence of P’s. The test was 


expected to be a success. Our expectation was confirmed. 


P PQ R 








P; incrTime(5);Q; incrTime(10) P; incrTime(31) (timeout 
has occurred); R. The test was expected to be a success. 


Our expectation was confirmed. 
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P; incrTime(5); Q; 





incrTime (31) (timeout has occurred); 


P; incrTime(5); QO; incrTime(5); R; incrTime(21) (timeout has 





occurred). We tested for multiple intervals in this test and 


the next to ensure that the assertion would observe more 











than a single time interval. We expected success. Our 
expectation was confirmed. 
8. Whenever P Then Q or Rot R Within T 
Whenever P then Q or not R within T TRTimeoutSimulatedTime timer = new 
N, PLY TRTimeoutSimulatedTime(30, thé); 
iitimeoutFire (transition) 
“PL 
a & \ ti Fit 
@ ee. init =| —+> Cc Pp 4, timeoutFire[]’ O S 
On-Entry/timer restart); 
i 
SY 4 f ewe 
Ny REI 
O Error 
On-Entry/bSuccess = false, fm EI RE] 
P; incrTime (31) (timeout has occurred). We expected 


obvious success. Our expectation was confirmed. 
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Q; incrTime(31) (timeout has occurred). We expected 
success. Our expectation was confirmed. 


R 








R; incrTime (31) (timeout has occurred). We expected 


success. Our expectation was confirmed. 


P Q 





30 


P; incrTime(5); Q; incrTime(26) (timeout has occurred). 


We expected success. Our expectation was confirmed. 


P R 





P; incrTime(5); R; incrTime(26) (timeout has occurred). 


We expected failure. Our expectation was confirmed. 
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P; incrTime(5); QO; incrTime(5); R; incrTime (21) (timeout 





has occurred). We expected failure. Our expectation was 


confirmed. 





P; incrTime(15);P; incrTime(5); QO; incrTime (26) (timeout 





has occurred); R. Our goal in this test and the next was to 





ensure that the assertion could handle overlapping time 
intervals and that the assertion observes more than the 


first P in a sequence of P’s. The test was expected to be a 


success. Our expectation was confirmed. 


P PQ R 





P; incrTime(5);0; incrTime (10) P; incrTime (21); R; 





incrTime(10) (timeout has occurred). The test was expected 


to be a failure. Our expectation was confirmed. 


74 





P; incrTime(5); Q; incrTime(26) (timeout has occurred); 
P; incrTime(5); R; incrTime(26) (timeout has occurred). We 
tested for multiple intervals in this test to ensure that 


the assertion would observe more than a single time 





interval. W xpected failure. Our expectation was 
confirmed. 
9. Whenever P Then Q and Not R within T 


TRTimeoutSimulatedTime timer = new 
TRTimeoutSimulatedTime(30, this}; 
}itimeoutFire (transition) 


Whenever P then Q and not R within T 


\, PLY 


t | Nx ROS 
init Pp 
@-F 9, 0 
On-Entry/timer.restart(); O Error 


a timeoutFire[]’ 
\, Oy 


Oc 
Ny RIV 


\, timeoutFire[} 


O Success 


75 





P; incrTime (31) (timeout has occurred). We expected 





obvious failure. Our expectation was confirmed. 








Q; incrTime (31) (timeout has occurred). We expected 


success. Our expectation was confirmed. 


R 








R; incrTime (31) (timeout has occurred). We expected 


success. Our expectation was confirmed. 





P; incrTime(5); Q; incrTime(26) (timeout has occurred). 


We expected success. Our expectation was confirmed. 
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P; 


incrTime(5); R; incrTime (26) 


(timeout has occurred) 
We expected failure. 


Our expectation was confirmed. 





P; 


incrTime (5); QO; incrTime(5); R; 
has 





incrTime (21) (timeout 
occurred). We expected failure. Our expectation was 
confirmed. 
P Q Q 





P; 


incrTime (5); QO; incrTime (5); Q; incrTime (21) 
(timeout has occurred) 





We expected success. 


Our expectation 
was confirmed. 


PQQP Q 





P; anerTime(5); QO; %incrTime(5); QO; incrTime(5); P; 
incrTime (20); OQ; incrTime(15) (timeout has occurred). Our 
goal 


in this test was to ensure that the assertion could 
handle overlapping time intervals 





and that the assertion 
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observes more than the first P in a sequence of P’s. The 





test was expected to be a success. Our expectation was 


confirmed. 


PQQ PQ 








Py incrTime (5); QO; incrTime (5); QO; incrTime (21) 


(timeout has occurred); P; incrTime(5); QO; =<aincrTime(26) 








(timeout has occurred). We tested for multiple intervals in 








this test to ensure that the assertion would observe more 





than a single time interval. We expected success. Our 





expectation was confirmed. 


B. ADDITIONAL ASSERTION DIAGRAMS UNBOUNDED BY TIME 


We felt that this assertion statement should be 


separated from the other assertion statements becaus th 








time is unbounded. It still has merit as an assertion 





statement but may not be as useful as the other assertions. 
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1. Whenever P Then Not Q After T 


TRTimeoutSimulatedTime timer = new 
Whenever P thennot Q after T TRTimeoutSimulatedTime(30, this}; 
{itimeoutFire (transition) 


\ 
@ Orit 2 PLY Op & O}success 


——_ 
On-Entry/timer.restart(; -\, timeoutFire[}’ 


| | \, oy 


~ 
~ OY (3 Error 


On-Entry/bSuccess = false; 








P; incrTime (31) (timeout has occurred). We expected 


obvious success. Our expectation was confirmed. 


Q 





OQ; incrTime (31) (timeout has occurred). We expected 





success. Our expectation was confirmed. 





P; incrTime(5); Q; incrTime(26) (timeout has occurred). 


We expected success. Our expectation was confirmed. 


719 





P; incrTime(31) (timeout has occurred); Q . We expected 





failure. Our expectation was confirmed. 





P; incrTime (10); P; incrTime (10); QO; 





incrTime (21) (timeout has occurred). Our goal in this test 


was to ensure that the assertion could handle overlapping 





time intervals and that the assertion observes more than the 





first P in a sequence of Ps. We expected success. Our 


expectation was confirmed. 


PP Q 








P; incrTime(5); P; incrTime(31) (timeout has occurred); 


Q. We expected failure. Our expectation was confirmed. 
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P; 


P; incrTime(31) (timeout has occurred); OQ. 


incrTime(5); Q; 


incrTime (26) (timeout has occurred); 








multiple 


We tested for 
intervals 








would observe more than a single tim 





failure. 


in this test to ensure that the assertion 





interval. We expected 
Our expectation was confirmed. 
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